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Introduction 


Completing a difficult task requires a combination of skill and 
will. For example, while young athletes may have all of the 
will they can muster to become professionals, they still need 
to focus on skill development and growth to achieve their 
goal. The use of research and evidence to inform decisions in 
schools and districts is a surprisingly parallel story. 


Athletes put in a significant amount of 

work to achieve their goals. Why? Research 
suggests that intrinsic motivation provides the 
foundation on which athletes, in collaboration 
with their coaches and teammates, build and 
develop their professional skills (Lewthwaite 

& Wulf, 2017). To get the ball rolling toward 
becoming a professional athlete, there needs 
to be a foundational will to put in the work. 


Similarly, to get the ball rolling toward more 
widespread use of research and evidence in 
K-12 education decision-making, we must 
first determine whether the will exists. The 
National Center for Research in Policy and 
Practice recently revealed important insights 
about the willingness of education decision- 
makers to use research and evidence (Penuel 
et. al., 2016). In a study with 733 different 
school and district leaders, most education 
leaders self-reported positive attitudes 
when asked about the value of research 
(Ibid). Moreover, most education leaders 
also self-reported regular use of research 
and evidence to inform decisions (Ibid). 


Professional athletes demonstrate that will and 
motivation are essential to achieving success. 
However, skill is still required to succeed. 
Athletes deeply focus on skill development, 
particularly by collaborating with and learning 
from experts in their field, such as coaches. 

In a recent study, Jowett (2017) found that, 


while coaching “has been often viewed as 

a context within which coaches operate 

to largely bring about changes in athlete's 
performance and wellbeing,” coaching 
effectiveness is tied to the quality of coach- 
athlete relationships. This implies that in order 
for athletes to develop and improve, they must 
have an open dialogue and a trusting, ongoing 
relationship with their expert coach. Together, 
coaches and athletes improve practice. 


Similarly, recent evidence demonstrates that 
education leaders have the motivation to 
make research and evidence-based decisions, 
yet the skill development process is less 
clear. Education leaders do not have defined 
coaches to support them in improving their 
data-driven decision-making practices. Few 
studies have been able to push beyond the 
veil of self-reporting by district and school 
leaders to understand the ways in which 
individuals within districts actually use research 
and evidence to inform decision-making; 

as of yet, researchers have just begun to 
measure the quality and rigor of research and 
evidence use by education leaders (Coburn 

& Turner, 2011). Most studies suggest limited 
use of research and evidence in education 
leadership roles (Coburn & Turner, 2012). 
Given the clear desire to make research and 
evidence-based decisions, it is likely there 

is a lack of skill to successfully do so. Even 
among the 733 education leaders involved 
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in the recent study conducted by Penuel et 

al. (2016), the researchers found that few 
leaders consulted research when confronted 
with a challenge. Moreover, researchers 
found that education leaders’ interpretations 
and conclusions based on research were 
frequently inappropriate and inaccurate (Ibid). 


Together, these findings suggest there 
is a desire among education leaders to 
use research and evidence to inform 
decisions. However, like athletes, 
education leaders need support from 
experts in developing capacity and skill. 


Support for Education Leaders 


In order to work toward increasing districts’ 
and schools’ capacities to effectively use 
research and evidence to guide and inform 
decisions, it's valuable to first understand 

the challenges that make skill development 
difficult. Studies have found that the use of 
evidence to develop and inform decisions 

is limited by staff capacity (including time), 
research skills, and support, as well as 
organizational culture and an institution's 
historical relationship with research (Marsh, 
Pane, & Hamilton, 2006). Many districts do not 
have the internal capacity and research skills 
to devote to analyzing and interpreting data; 
often, this is because most districts are small 
and have limited staff numbers within their 
administration (National Center for Education 
Statistics, 2011). This limited capacity can lead 
to cases where claims are based on statistically 
insignificant or inappropriately interpreted 
findings (Marsh, Pane, & Hamilton, 2006). 


Digital Promise’s Marketplace Initiative, which 
has collaborated with district leaders across the 
country to develop tools and resources that 
support their edtech evaluation efforts, wanted 
to understand how these tools help support 
education decision makers use evidence to 
inform procurement and adoption decisions. 
In 2018, Digital Promise administered a 

survey to further contribute to the research 
that exists on research and evidence use 

in schools and districts. Most research on 

this topic only considers the perspectives 

of superintendents and principals. This new 
survey, however, intended to incorporate 
perceptions and attitudes around research 


and evidence use across a more expansive 
list of education roles, including educators, 
coaches, and instructional specialists. 


To guide our future work, the survey 

also sought to understand self-reported 
research and evidence use across Digital 
Promise networks to understand who needs 
coaching and support going forward and 
in what areas. Through the survey, findings 
uncover differences between education 
leaders’ perceptions of the significance of 
research and evidence to inform program 
adoption and procurement. Moreover, the 
survey finds interesting differences in the 
types of research and evidence used by 
various education roles to inform decisions 
around the adoption and procurement 

of innovative educational programs. 


In addition to understanding the significance 
and types of evidence education leaders 
and practitioners use to make decisions, 
this survey sought to learn more about a 
piloting tool developed by Digital Promise 
to support education leaders and educators 
in conducting pilots around education 
technology and innovative programs, 

the Edtech Pilot Framework. The survey 
intended to learn about who accesses the 
Digital Promise Edtech Pilot Framework and 
to understand their motivations to use the 
Framework. Further, the survey intended 

to uncover potential correlations between 
self-reported research and evidence use 
and toolkits developed by Digital Promise, 
such as the Edtech Pilot Framework. 
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Methodology 


In June 2018, Digital Promise administered a 12-question 
survey through SurveyMonkey, which was completed by 143 
individuals in the education field. 


Most respondents came from Digital 
Promise’'s Research and Evidence Use in 
Edtech community on edWeb. Digital Promise 
launched this community in the fall of 2017 
to engage educators, education leaders, 
researchers, and product developers in 
discussions around strategies to effectively 
and appropriately use research and evidence 
to inform decisions. The second network we 
administered the survey to was the League 
of Innovative Schools, a national network 
organized by Digital Promise of more than 
100 forward-thinking education leaders 
who are working together to improve 
outcomes for students and solving the 
challenges facing K-12 schools. Researchers 
administered the survey to these two 
networks to understand and compare 

their use of research and evidence. 


The survey required respondents to note 
their role/occupation in education and 
whether they are members of the League 

of Innovative Schools, which enabled 
researchers to make interesting comparisons 
in the analysis. In addition, the survey asked 
respondents, “How significant a role does 
research and evidence play at your school 

or district in decisions about procurement 
and program adoption?” with the option 


Key Player Profile 


Researchers sought to learn from 
superintendents and principals to understand 
their responses compared to the available 
literature. However, researchers also targeted 
teachers and school staff, such as coaches, 
instructional and educational technology 


to answer on a scale from 1, meaning no 
role, to 10, meaning an essential role. 


Respondents were also asked to note the 
types of evidence they use to inform program 
adoption decisions, with options to choose 
from including pilots conducted by their own 
school or district, pilots conducted in another 
district, recommendations from colleagues, 
research provided by the developer, and 
“other” to include any other type of evidence. 
Finally, in an effort to understand a potential 
correlation between research and evidence 
use and the Edtech Pilot Framework, 
respondents were asked about their familiarity 
with the tool, why they visited the site, and 
how useful the site was in meeting their needs. 


Survey responses were coded to equate to 
numeric values. For example, when asked 
whether a respondent was familiar with 
Digital Promise’s Edtech Pilot Framework, 
“no” responses were coded to equal 0 while 
“yes” responses were coded to equal 1. After 
coding the responses, analysis was completed 
using IBM SPSS Statistics software version 
25. The analysis included descriptive and 
frequency statistics, as well as independent 
sample T-tests, when appropriate. 


specialists, and librarians, to give voice to 
their use of research and evidence. The survey 
was administered and open throughout the 
month of June 2018. Through each effort to 
administer the survey, potential participants 
were notified that 10 respondents selected 
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at random would receive a $25 gift card to education institutions completed the survey. 


Amazon. Researchers requested members of Respondents represent an expansive view 
edWeb to complete the survey through an on K-12 decision-making across the United 
email sent by the edWeb team, as well as at States, with representation from 35 states, 
the end of a webinar hosted on June 13, 2018. in addition to four respondents involved 
League members received two emails in June in K-12 education outside of the U.S. 


2018 that asked them to complete the brief 
survey to help inform Digital Promise’s work. In 
total, 143 individuals responded to the survey. 


Educators 
Of those who completed the survey, 95 School Staff 
percent of respondents were directly involved 
in K-12 education decision-making (see Figure District 


1). The largest portion of respondents were Administrators 
educators, followed by school staff, which 
included roles like librarians, instructional and/ 
or technology specialists, paraprofessionals, 
program coordinators, and coaches. About 

13 percent of respondents were district 


administrators, including superintendents, 


School Leaders 


F ae Figure 1: 
and about eight percent were principals 
and school leaders. Additionally, seven Distribution of Survey Respondents 
individuals who work for education technology by Education Role 
companies, education nonprofits, or higher [Survey Question: What is your role?] 


Results 


Research and Evidence are Important in 
Decision-Making 


© Overall, education leaders’ responses e Survey results indicate that the types 
aligned with the findings of Penuel et of research and evidence used to 
al. (2016); education leaders reported inform decisions varies by role. 
that research and evidence play a 
significant role in decision-making. Survey results demonstrate that among those 
who responded across 35 states, research and 
@ On average, all respondents in K-12 evidence play a significant role in adoption 
education roles reported a mean of and procurement decisions for innovative 
7.45 when asked to rate the significance programs. Among all of the participants 
of the role of research and evidence employed in K-12 education settings, nearly 
in decision-making in their school or two-thirds rated research and evidence 
district on a scale from one, meaning no as a playing highly important or essential 
role, to 10, meaning an essential role. role in decision-making (see Figure 2). 
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Level of Significance Research and Evidence Play in K-12 Program 


Adoption Decisions (N = 136) 


On average, respondents in K-12 education 
settings reported that research and evidence 
play a moderate to highly important role in 
decisions about program procurement and 


adoption, though responses varied by role (see 


Table 1). When asked how significant a role 
research and evidence play at respondents’ 
schools or districts in decisions about 
procurement and program adoption, school 
leaders reported a statistically significantly 
higher score, an average of 8.20, compared 
to educators, an average of 7.45 (p<.10). 


This likely indicates an important discrepancy 
between leadership's intention and actual 
implementation. The finding may suggest 
that the top-down nature of decision-making 
processes in schools often leaves teachers 
out of important conversations around the 
goals of implementing innovative programs 


and the basis for procurement and adoption 
decisions. It may also indicate that, while 
principals and school leaders intend to make 
decisions based on research and evidence, 
educators disagree that their leaders’ skills 
lead to appropriately interpreted conclusions 
and, thus, decisions. Studies suggest that 
education leaders have proposed policies 
based on students’ performance ona single 
test, without appropriately accounting for the 
multitude of outside factors that influence 

a student's performance (Marsh, Pane, 

& Hamilton, 2006). In turn, the common 
misuse of data by education leaders can 
foster a mistrust of research and data among 
practitioners, which may be the basis for 

this statistically significant difference. 
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Role N Mean Std. Deviation 


Overall 132 7.45 2.34 
Educator 78 7.45 2.33 
Superintendent or 

District Administrator 18 Toe eto 
School Staff 26 7.04 2.47 
Principal or School Leader 10 8.20 1.32 


Table 1: 

Average responses rating how significant a role research and evidence play 
at respondents’ schools or districts in decisions about procurement and 
program adoption, by role on a scale of 1 (no role) to 10 (essential role). 


When asked about the types of evidence or district as the most common form of 

used to inform program adoption evidence, followed closely in frequency by 
decisions, respondents most frequently recommendations from colleagues. However, 
cited pilots conducted by their school responses varied by role (see Figure 3). 


100 


Pilots conducted by 
my school or district 


Pilots conducted in 
another district 


Recommendations 
from colleagues 


Research provided 
by the developer 


Other 


Educator District School Staff Principal or 
Administrator school leader 


Figure 3: 
The Types of Evidence Used to Inform K-12 Program Adoption Decisions 
by Role (N=136) 
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District and school leadership reported 
fairly different types of evidence that inform 
their program adoption decisions. About 83 
percent of district administrators reported 
using pilots conducted by their school or 
district as a form of evidence used to make 
program adoption decisions, compared to 
only 55 percent of school leaders. On the 
other hand, more than 80 percent of school 
leaders reported using research provided by 
the developer to make decisions, compared 
to only 33 percent of district administrators. 


Educators use a variety of evidence 
and research to inform their decisions. 
The majority of educators use pilots, 


about half use recommendations from 
colleagues, and far fewer use evidence 
gathered by other schools or districts. 


These findings illustrate the lack of 
collaboration in decision-making across 

the various roles in K-12 education. Likely, 

this finding suggests that district and school 
leaders should communicate more regularly 
with one another as well as educators to 
define types of evidence that should support 
program adoption and procurement decisions. 


Digital Promise’s Contributions to Evidence-Based 


Decision Making 


Members of the League of Innovative 

Schools included in this survey value the 

role of research and evidence and evidence 

in decision-making significantly more than 
non-League members surveyed. This result 
may be, in part, due to their work with 

Digital Promise. For instance, the Edtech 

Pilot Framework has supported members of 
the League of Innovative Schools in more 
regularly accessing, collecting, and using 
research and evidence to inform adoption 
and procurement decisions around innovative 
programs and products. When asked how 
significant a role research and evidence play at 


respondents’ schools or districts in decisions 
about procurement and program adoption, 
superintendents in the League of Innovative 
Schools reported a statistically significantly 
higher score, an average of 8.21 out of 10, 
compared to respondents not affiliated 

with the League of Innovative Schools, who 
reported an average of 7.05 (p<.10). These 
results are promising for Digital Promise’s 
research and evaluation efforts since they 
suggest that League leaders may value and use 
research more than non-League members. 
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Discussion 


Education Decision Makers Need Support 


Skill Building 


This survey echoes the results of the 

National Center for Research in Policy and 
Practice’s recent study (Penuel et al., 2016) in 
suggesting that there is a strong desire to use 
research and evidence, as well as thorough 
recognition of the need to do so. This survey 
highlights that research and evidence play 

a moderate to critical role in the majority 

of K-12 decisions around the adoption and 
procurement of innovative programs. 


These findings also imply that educators are 
interested in research and evidence, especially 
piloting, to inform their program adoption 
decisions. In order to support educators in 
collecting relevant data to inform decisions, 

it is important for organizations, including 
Digital Promise, to continue to provide 
professional learning opportunities along 

with free tools and resources to support 
educators in conducting pilot studies. 


Our theory of action is to be capacity builders. 
In 2014, in collaboration with Johns Hopkins 
University, Digital Promise conducted 

a study to understand the process K-12 
education leaders follow to make education 
technology (edtech) procurement decisions. 


Limitations 


The limitations of this survey mirror the 
limitations of this work across the field; self- 
reporting is not an accurate assessment of 
actual research and evidence use to inform 
decisions, much less the quality of that use. In 
fact, the discrepancy between school leaders’ 
and teachers’ reported level of significance 

of research and evidence use in decision- 
making further perpetuates the theory that the 


This study identified that K-12 education 
leaders conducted pilots as a reliable and 
relevant source to make these decisions. The 
survey's results also align with these findings, 
demonstrating that piloting continues to be 
the most common type of evidence used 

to inform these decisions. As a follow-up, 

in 2015, Digital Promise conducted a study 
with the University of California at Davis to 
identify best practices in edtech piloting; 
based on these findings, Digital Promise 
created the Edtech Pilot Framework to help 
education leaders and educators build their 
skills in conducting, interpreting, and using 
research and evidence to inform decisions 
that impact students. The Edtech Pilot 
Framework includes nearly 150 open-source 
tools and resources to support educators 
throughout the process of conducting pilot 
studies. In addition to conducting research 
and creating the Edtech Pilot Framework, 
Digital Promise has also conducted 45 pilot 
studies in partnership with K-12 school 
districts and led more than 20 webinars to 
support education practitioners in conducting 
edtech pilots. All of these efforts aim to build 
districts’ capacity to make evidence-based 
procurement and adoption decisions. 


will to use research and evidence to inform 
decisions is not matched with an appropriate 
skill level to accurately analyze and interpret 
evidence and research. Additionally, this 
survey involved a small sample size and 
those who did complete the survey did so 
voluntarily, which may mean they are more 
likely to indicate a desire to use research. 
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Future Studies and Support 


Like those inspired to become professional 
athletes, the results of this survey suggest 
educators and education leaders express a 
strong desire to use research and evidence to 
inform decisions about edtech and innovative 
programs. Unlike athletes, though, there is 
no clear coach or expert to support districts, 
schools, or individuals in acquiring the 

skills to analyze and interpret information 

to drive decision-making. Over the past 

five years, Digital Promise has engaged in 
capacity building efforts to support K-12 
school districts across the United States in 
accessing and using research and evidence 
to make program and product selection, 
adoption, and procurement decisions. 


Digital Promise is committed to further study 
of research and evidence use in education 

so we can better support research and 
evidence use in innovative program design, 
selection, adoption, and implementation. 
Digital Promise will continue work alongside 
educators and education leaders to provide 
professional learning opportunities to build 
education leaders and educators’ skills to use 
research and evidence to inform innovative 
program adoption and procurement 
decisions, thus allowing for the continuous 
improvement of programs that impact 
students’ learning and school experiences. 
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Appendix B: Decision-Making Around Edtech and 
Innovative Programs Survey 


1. What is your role? 
Superintendent 

District Administrator 
Principal or School Leader 
Educator 

School Staff 


Entrepreneur 


mp ano p 


g- Other (please specify) 
2. In which state is your school or district? 


3. How significant a role does research and evidence play at your school or 
district in decisions about procurement and program adoption? 


a. 1(no role) > 10 (essential role) 


4. What types of evidence do you use to inform your program 
adoption decisions? (Choose all that apply) 


a. Pilots conducted by my school or district 
b. Pilots conducted in another district 

c. Recommendations from colleagues 

d. Research provided by the developer 

e. Other (please specify) 


5. Are you familiar with the Digital Promise Edtech Pilot Framework, 
a free resource to support the pilot process? 


a. Yes 
b. No 
6. Have you visited the Digital Promise Edtech Pilot Framework? 
a. Yes 
b. No 
7. Why did you visit the Digital Promise Edtech Pilot Framework? (Check all that apply) 
To find tools and resources to pilot innovative programs 
To find free tools and resources about innovative programs 
To learn about piloting 
Colleagues recommended it 
To get information about piloting Micro-Credentials 
To view video case studies 


To read Pilot Study Briefs, reports on pilots conducted in other districts 


soe =o aos» 


Other (please specify) 
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8. To what extent do you agree with the following statement: | found 
the Digital Promise Edtech Pilot Framework useful. 


a. Strongly disagree 
b. Disagree 
c. | don't know 
d. Agree 
e. Strongly agree 
9. Have you heard of Digital Promise’s Pilot Study Briefs? 
a. Yes 
b. No 
10. Have you read one or more of the Pilot Study Briefs? 
a. Yes 
b. No 


11. How useful was the information you read in the Pilot Study Brief? 
a. Not useful at all 
b. Slightly useful 
c. Moderately useful 
d. Very useful 
12. Is your district a member of the League of Innovative Schools 
a. Yes 
b. No 


c. | don't know 
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